Skip to content

[Feature] Let the complexity signal consume a remote backend (score.v1 + label_distribution.v1) - #3542

Open
theohsiung wants to merge 11 commits into
vllm-project:mainfrom
theohsiung:feat/2921-complexity-backend
Open

[Feature] Let the complexity signal consume a remote backend (score.v1 + label_distribution.v1)#3542
theohsiung wants to merge 11 commits into
vllm-project:mainfrom
theohsiung:feat/2921-complexity-backend

Conversation

@theohsiung

@theohsiung theohsiung commented Sep 7, 2026

Copy link
Copy Markdown
Collaborator

Summary

Complexity could not name a remote model at all — CategoryModel was the only
struct with a Backend field. It now reads both remote shapes, because a
difficulty model can be built either way and the contract decides how the
runtime reads the answer:

  • score.v1 (new) — a regression model returns one number; each rule
    converts it through its own boundaries. One request, one call, however many
    rules read it. Implements the ScoringBackend tier, declared since the
    three-tier split with a note saying nothing wired it in.
  • label_distribution.v1 (reused) — a three-class model returns
    hard/easy/medium, so the winning label is the verdict and its
    probability is a real confidence.

Both go through the shared connector, so timeout, retry, byte caps and error
mapping stay shared.

Decisions worth knowing

  • Backend attaches to the module, not a rule: routing.signals is replaced
    wholesale per recipe, so a rule-level backend would vanish under any recipe
    that did not repeat it.
  • Direction lives in the field names (hard_above+easy_below, or
    hard_below+easy_above), so there is no direction/min/max field and
    an overlapping band cannot be written. threshold stays the symmetric
    shorthand.
  • contract is required for complexity: with two readable shapes, guessing
    wrong would surface per request rather than at config load. Category, reading
    one, keeps the default.
  • score.v1 reports no confidence — a score just short of the boundary is
    the least certain position, not a strong one. Validation warns, since it
    changes decision ranking.

Full reasoning is in #2921's body.

Validation

make test-semantic-router, recipe-conformance-static, website build, and a
new E2E profile run green on kind. The profile declares no local candidates, so
a verdict can only come from the remote call, and one score reaches two
different verdicts through two rules' boundaries.

make dashboard-check fails on three evaluationplane tests — verified
pre-existing at origin/main; a bare go test silently skips them and reports
ok.

The E2E model is general-expert because the shared AIGatewayRoute matches
x-ai-eg-model against six exact names. That is also why
category-remote-backend has failed since #3138: filed as #3543.

Closes #2921

@netlify

netlify Bot commented Sep 7, 2026

Copy link
Copy Markdown

Deploy Preview for vllm-semantic-router ready!

Name Link
🔨 Latest commit bbed0bc
🔍 Latest deploy log https://app.netlify.com/projects/vllm-semantic-router/deploys/6a9ed2e1904b230008b582d0
😎 Deploy Preview https://deploy-preview-3542--vllm-semantic-router.netlify.app
📱 Preview on mobile
Toggle QR Code...

QR Code

Use your smartphone camera to open QR code link.
🤖 Make changes Run an agent on this branch

To edit notification comments on pull requests, go to your Netlify project configuration.

The shared backend block reached only the category signal: `CategoryModel`
was the one struct with a `Backend` field, and the contract whitelist held
one value. Complexity could not name a remote model at all, which is the
gap vllm-project#2921 exists to close.

Complexity accepts two response shapes, because a difficulty model can be
built either way, and the contract chosen decides how the runtime reads
the answer. `score.v1` carries a continuous score; the existing
`label_distribution.v1` carries the verdict as a label. Neither
substitutes for the other, so this widens the resolver to take every
contract a consumer declares. Passing exactly one keeps it available as
the default for an omitted field - category is unchanged - while a
consumer accepting several cannot default: guessing wrong would surface
per request rather than at config load.

The backend attaches beside `prototype_scoring` rather than on a rule.
`routing.signals` is replaced wholesale per recipe, so a backend declared
there would vanish under any recipe that did not repeat it, leaving a
signal that still runs but has quietly reverted to local.

A score in the model's own units cannot be compared against `threshold`,
which is symmetric because the local margin is signed and centred on
zero. A rule can now state its two boundaries explicitly, and the pair it
uses says which way difficulty runs: hard_above with easy_below where a
higher score is harder, hard_below with easy_above where a lower one is.
Encoding direction in the names keeps a separate direction field out of
the schema and makes an overlapping band impossible to write. `threshold`
stays valid as the symmetric shorthand.

The reference config gains a rule using the explicit pair, since the
asymmetric form is useful to the local path too. The lower-is-harder pair
is excepted from reference coverage instead: the local margin is
hard-minus-easy, so a higher value is harder by construction, and a local
example would document the wrong direction.

Signed-off-by: theohsiung <theobear870924@gmail.com>
ScoringBackend has been declared since the three-tier backend split but
carried a note saying nothing implemented or wired it. score.v1 is its
first consumer.

It cannot reuse HTTPClassifierInference. That type exists to align a
label distribution onto a mapping, so it requires one with at least two
labels; score.v1 has no labels at all, the score being the whole
product. This is a separate, thinner reader over the same connector,
which keeps timeout, retry, byte caps and error mapping shared rather
than duplicated.

The wire shape is the HuggingFace text-classification array, because
score.v1 shares the http_classify protocol and endpoint - a regression
head deployed behind the usual /classify needs no shim. The label is
meaningless here. Anything other than exactly one entry leaves "which is
the score" undefined, so it fails rather than picking one, and the value
is returned unclamped because a regression head is not a probability.

Score takes the caller's context. The interface had none, which is the
gap the category backend had to close after the fact; starting with one
avoids repeating it. Nothing implemented the interface, so widening it
breaks no caller.

Signed-off-by: theohsiung <theobear870924@gmail.com>
The direction pair was structurally validated - no overlap, no mixed
directions, no half-declared pair - but nothing tied it to where the
score comes from. A local rule could declare hard_below/easy_above and
be accepted, then invert every verdict it reached.

The local margin is hardScore minus easyScore, so a higher value is
harder by construction, not by convention. A local rule declaring the
opposite direction would score the user's own hard examples as easy,
and the config would look entirely reasonable while doing it. The
inverse is already expressible locally by swapping the candidate lists,
so the pair is rejected there rather than honoured.

Rules inside a recipe are checked too. routing.signals is replaced
wholesale per recipe, so a rule that exists only in one would otherwise
skip validation entirely - the same reason the backend itself is
declared on the module.

Found by asking what actually stops the semantically wrong combination,
after excepting those two fields from reference coverage for exactly
this reason and not carrying the conclusion into validation.

Signed-off-by: theohsiung <theobear870924@gmail.com>
Both remote contracts now produce the complexity signal, and the local
prototype path is unchanged when no backend is configured.

The two paths return the same rule-result shape as the local classifier,
so the publish loop stays shared: match names, metrics and published
values cannot drift between local and remote because there is only one
place that writes them.

score.v1 makes one call per request, not per rule. The score is a
property of the request; a rule differs only in where it draws its
boundaries, which is why the backend is declared on the module and why
five rules over one model still cost one call. label_distribution.v1
reads the verdict straight off the winning label, consulting no
boundaries at all, and reuses the shared sequence backend with the
verdict vocabulary declared inline as the generic classifier signal
already does.

Rules still matter on the label path even though every rule receives the
same verdict, because each carries its own composer and can be gated
differently.

ConfidenceReported separates a real confidence from the absence of one.
The local path and the label contract both report one; score.v1 does
not, so the publisher leaves the key absent and the decision engine
falls back to its structural default while marking the pool unscored -
the treatment keyword, language and pii already receive. Writing a zero
instead would silently rank the decision last among scored competitors.

The request context reaches the signal, so an abandoned request cancels
the outbound call. That meant threading it through the request-fact
dispatcher, which had none while the primary dispatcher already carried
one.

Signed-off-by: theohsiung <theobear870924@gmail.com>
pkg/config/AGENTS.md requires a config-contract change to update the
public docs in the same change, so the tutorial, installation
configuration, config/README.md and the v0.3 proposal all describe the
new surface rather than leaving the schema ahead of its documentation.

Two consequences are correct but easy to miss, so the Router says them
at startup instead of leaving them to be discovered:

score.v1 reports no confidence, so a decision gated on one of those
rules ranks on the engine's structural default rather than a reported
score - while a local-backed rule beside it still reports one, making
the change invisible in the config but visible in which decision wins.
label_distribution.v1 has no such gap, and the advisory says so.

The hard/easy candidate lists are how the local path produces a score.
With a backend they are never read, so editing them has no effect at
all; a config mid-migration will legitimately still carry them, which
is why this is an advisory rather than an error.

The advisories are returned as strings and logged by the validator
rather than logged directly, so the reasoning is testable without
capturing log output - this package has no precedent for asserting on
log lines.

Signed-off-by: theohsiung <theobear870924@gmail.com>
**The local path lost its confidence (a regression).** ConfidenceReported
was introduced for score.v1 but only ever set on the remote label path, so
every config with no backend at all stopped publishing
SignalConfidences for complexity: confidence-based decision ranking,
numeric predicates on complexity conditions, projection inputs and
router-learning outcome scores all silently changed. The field's own doc
comment asserted the opposite of what the code did.

**The local path ignored the new boundary pair.** It still called
classifyComplexityDifficulty(rule.Threshold, ...) and never resolved
EffectiveBoundaries, so a rule declaring hard_above/easy_below ran with
threshold zero - every positive margin hard, medium unreachable, the
declared cut points discarded. The reference config shipped exactly that
rule, with a comment claiming it worked. Both paths now share one
comparison, resolved once at construction so a malformed pair fails at
startup rather than per request.

**The validator never ran at config load.** Category registers its
contract validator in globalConfigContractValidators; complexity had no
equivalent entry, so validate-config tooling, the apiserver and the DSL
paths accepted an unresolvable backend and the error only surfaced at
classifier construction. The comment claiming otherwise is now true.

**threshold was accepted under score.v1.** A remote score arrives in the
model's own units, so a symmetric threshold cannot convert it: against a
[0,1] scorer it puts everything past the hard cut and makes easy
unreachable, while a rule declaring nothing collapses both cut points
onto zero. score.v1 now requires the explicit pair.

**The DSL dropped the boundary pair.** The decompiler never emitted the
four fields and the compiler never read them, so YAML -> DSL -> YAML
reverted a rule to threshold semantics. Reading them back through
getFloat64Field rather than float32 keeps a declared 0.85 from becoming
0.8500000238418579 - caught by the round-trip test itself.

Signed-off-by: theohsiung <theobear870924@gmail.com>
AGENTS.md requires E2E coverage for a behaviour-visible config change, and
the gate assigns this one the envoy-ai-gateway profile, which never
touches the remote scorer.

The profile declares no hard/easy candidates at all. With a backend the
local prototype path is unreachable anyway, so a verdict reaching the
router can only have come from the remote call - that is what makes the
assertion attributable rather than merely consistent. The test then pins
two scores and requires them to route differently: 0.90 clears
needs_reasoning's boundary but sits inside extreme's medium band, while
0.99 clears both. One remote call, two rules, two verdicts is a shape
prototype scoring cannot produce here.

The mock scorer is an inline Python server in a ConfigMap on
python:3.12-alpine, following mock-category-classifier, so the profile
builds no image of its own. It returns exactly one entry because score.v1
rejects anything else, and an unmarked request returns a constant so an
assertion cannot go flaky for reasons unrelated to the router.

Two things this cost, both worth recording:

The router OOMKilled at 2Gi, copied from the category profile. Taking the
scorer off-box does not avoid loading the local mmbert embedding model the
response cache is configured with; the embedding models are gated
separately from the classifier backend. 6Gi, as the jailbreak-onerror
profile already found.

The model is general-expert rather than a profile-specific name. The
shared AIGatewayRoute matches x-ai-eg-model exactly against six names, so
anything else has no route and every request 404s at the gateway while the
router itself stays healthy and logs a correctly constructed backend -
which is exactly how it looks like a router bug and is not one.

Also releases the score backend's connector on Classifier.Close. A
classifier is rebuilt per recipe and again on every dynamic-config
reload, so one that is never closed leaks an idle HTTP transport each
time.

Signed-off-by: theohsiung <theobear870924@gmail.com>
@theohsiung
theohsiung force-pushed the feat/2921-complexity-backend branch from e155bdd to a33ac62 Compare September 7, 2026 09:44
@Xunzhuo

Xunzhuo commented Sep 7, 2026

Copy link
Copy Markdown
Member

Thanks for working on it!

@github-actions github-actions Bot added pr/needs-review Ready for reviewer attention. wg/router-models-inference-runtime Owned by the Router Models and Inference Runtime Workgroup. labels Sep 7, 2026

@Xunzhuo Xunzhuo left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the thorough contract work. Four integration blockers remain: remote complexity still constructs and preloads the unused local classifier before attaching the backend, so remote-only startup depends on local model resources and readiness ignores the backend; score.v1 treats a missing or null score as numeric zero; explicit boundaries accept NaN and Inf; and dynamic reload replaces RecipeClassifiers without lifetime-safe retirement, leaking the old remote connectors while only the startup set is registered for shutdown. Please bypass local construction in remote mode with backend-aware readiness, require score presence while preserving valid zero, reject non-finite boundaries, and make reload ownership safe for in-flight snapshots. The exact-head quality job also fails MD012 in the complexity tutorial.

@github-actions github-actions Bot added pr/blocked Blocked on a named decision, dependency, or required check. and removed pr/needs-review Ready for reviewer attention. labels Sep 7, 2026
md-fmt enforces MD012/no-multiple-blanks; the section I inserted left two
consecutive blank lines before the next heading.

Signed-off-by: theohsiung <theobear870924@gmail.com>
vllm-project#3489 renamed providers.defaults.default_model to model and added
provider to backend_refs. Merging main brought no textual conflict here,
because the profile is a new file - so the stale spelling would have
survived and providers.defaults.model would simply have parsed as empty.

Verified by loading the profile's config subtree through config.Load
rather than by reading the diff.

Signed-off-by: theohsiung <theobear870924@gmail.com>
@github-actions github-actions Bot added pr/needs-author Waiting for author changes or response. pr/blocked Blocked on a named decision, dependency, or required check. and removed pr/blocked Blocked on a named decision, dependency, or required check. pr/needs-author Waiting for author changes or response. labels Sep 7, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

pr/blocked Blocked on a named decision, dependency, or required check. wg/router-models-inference-runtime Owned by the Router Models and Inference Runtime Workgroup.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Feature] Define the score.v1 contract for complexity backends

2 participants